Augmented Skeleton Based Contrastive Action Learning with Momentum LSTM for Unsupervised Action Recognition

Action recognition via 3D skeleton data is an emerging important topic. Most existing methods rely on hand-crafted descriptors to recognize actions, or perform supervised action representation learning with massive labels. In this paper, we for the first time propose a contrastive paradigm named AS-CAL that exploits different augmentations of unlabeled sequences learn representations in unsupervised manner. Specifically, contrast similarity between augmented instances input sequence, which are transformed multiple novel augmentation strategies, inherent patterns (“pattern-invariance”) transformations. Second, encourage pattern-invariance more consistent representations, momentum LSTM, implemented as momentum-based moving average LSTM based query encoder, encode long-term dynamics key sequence. Third, introduce queue store encoded keys, allows flexibly reusing proceeding keys build dictionary facilitate learning. Last, Contrastive Encoding (CAE) represent human’s effectively. Empirical evaluations show our approach significantly outperforms by 10–50% Top-1 accuracy, and it can even achieve superior performance many (Our codes available athttps://github.com/Mikexu007/AS-CAL).